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© Advanced user interface. 

© An advanced user interface for use with a computer system operating on an integrated operating environ- 
ment The integrated operating environment allows a plurality of application programs to be running simulta- 
neously one of which is designated the active application program to which all input data is directed. The 
advanced user interface allows a user to select among user-friendly input devices to operate any application 
program according to his individual preferences without change to the application program code. The advanced 
user interface includes alternate input modules which translate the input signals transmitted from the various 
input devices into input messages useable by the rest of the interface. The advanced user interface also 
includes interface profiles which contain mappings of the input messages against corresponding commands 
useable by the application programs, the integrated operating environment or other modules of the advanced 
user interface itself. An environment link module refers to the interface profiles and matches the input message 
against the corresponding command for the application program active at the time the input signal was 
transmitted and send the corresponding command to that application program. 
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ADVANCED USER INTERFACE 



This invention relates to improvements in entering input data into digital computers. More particularly, it 
relates to an advanced user interface which allows a user to select one or more input devices to input data 
into a computer running a program originally written for a different input device. 

In the past computers were used only by scientists, mathematicians, and other high-level, sophisticated 
computer users. As computer technology progressed, and particularly with the advent of the persona 
computer, data processing has reached every level of society, and every level of user. The trend is for 
fewer computer users to be computer professionals or sophisticated in data process.ng techniques. Access 
to computers will increase even further in the future as computer hardware and software increase m power 

HowevT/despite the prevalence of computers in our society, many are apprehensive about using them. 
The traditional text display and alphanumeric keyboard input device are not readily learned requiring the 
memorization of the correct format of dozens, if not hundreds, of commands. Unless the user is a 
reasonably good typist, data entry can be inconvenient and time consuming. The hours of experiments ion 
which may be necessary to become proficient with a single piece of software becomes extremely frustrating 
for the unsophisticated user who is likely to avoid the effort altogether. 

It has therefore become necessary to design what have become known in the art as user friendly 
input devices and computer programs. Such "user friendly" devices and software are designed to allow an 
unsophisticated user to perform desired tasks without extensive training. One of the more popular ,nput 
devices is the mouse pointing device. The mouse generates signals correspond.ng to the cl.rect.on and 
distance it is moved across a flat surface by the user. The computer responds to th.s information by 
changing the position of the cursor on the computer display to allow the user to point to any displayed 
object. Once the cursor is correctly positioned, the user can perform a number of functions by depressing 
one or more buttons on top of the mouse. . 

Suman factor studies have shown that a device which allows the user to input data directly on the v.sua 
display screen of a computer, generally known in the art as a touch input device, *h.e^g^ 
immediacy and accuracy between man and machine. One of the first input devices for use at the d.splay 
surface was the light pen. The light pen is an optical detector in a hand held stylus, which is placed agsins 
the face of a cathode ray tube. The location of the light pen is determined by detecting the coordinates of 
the dot of light which is the scanning raster of the display. A second type of touch input device is a 
mec a ical deformation membrane which is placed over the display screen. The membrane is a transparen 
overlay which consists of two transparent conductor planes disposed on a flex.b.e surface^When a election 
is made the user mechanically displaces one of the conductor planes to touch the other by a fmger or 
stylus touch, thereby bringing the conductors into electrical contact with each other. Appropriate electronics 
Sare translate J electrical signals generated by the finger or stylus touch to the .ponton on he 
display surface. Another touch input device is a capacitive transparent overlay placed over the display 
s reen which includes transparent conductors driven by an electromagnetic signal. The input device can 
de ect 'the location of a finger touch by the change in capacitance of the overlay or, alternately, a stylus is 
Ts^reL the electromagnetic signals from the overlay back to the computer to dete = , the s^u 
position Yet another touch input device uses a frame which fits around the d.splay screen havng a number 
of Tared or visible light transmitters and receptors arranged in parallel horizontal and verted direction^ 
When the user's finger biocks- the light beams, the horizontal and vertica. receptors note the absence of the 
sionals thereby locating the position of the action desired by the user. 

9 Man" othell user friendly input devices which respond to actions of the user such as vo.ce recognitor, 
units or digital tablets, have been developed. }An/ j K , 
n addition to user friendly input devices, designers have made efforts to develop more user friendly 
software. One technique which has been employed is to provide the user with a menu of cho.ces of the 
paZlsr tasks or functions which can be performed. In this way, the user is not required to , eommri Hong 
sts o commands to memory. The menu can be a full or partial screen disp.ay with spaces adjacent to the 
menu entries to which the cursor is moved by keyboard or by other cursor moving dev.ce to select a 
particular action. Alternatively, the user can select an action by entering an alphanumeric character 
associated with the menu selection on a command line. IA / *h«,,* 

Another recent trend is to provide some sort of integration of computer program app cations. Without 
integration, the user must employ separate application programs for word process.ng database man pu.a- 
Z 9 l^c 9 nH ironic mail functions, and so forth. It is often quite difficult to mtegrate the outputs of 
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It is still yet another object of the invention to allow different users who ooerate a sinote rnmn„tor m 
construct different user interfaces to accommodate individual preferences I , inpu^ng data ' ' * 
These objects and others are accomplished by an advanced user interface which 'operates with an 
mtegrated operating environment capable of running a plurality of application programs and ^t h u S 2 ^ 

cL^Tir 6 f ° CUSin9 fUnCti ° nS Pr0Vid8d * the ™Lm£. The JZS user erf i 
comprised of three major modules, an alternative input subsystem module, an interface proS module and 
an environment link module. The alternative input subsystem module ^JcZm'Z^ between 2 
attached user friendly input devices and the remainder of the advanced user ISS^^^S^ 
programs through the integrated operating environment. The afternaBve ^^S^ZS^Z 
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The environment link module communicates with integrated operating environment and contains 
algorithms for the execution of the advanced user interface procedures. For example, it takes the input 
messages generated by the alternate input subsystem, queries the integrated operating environment as to 
which application program is active, matches the input messages to the corresponding commands in the 
appropriate application profile within the interface profile module, and initiates an action based on the set of 
instructions such as sending messages to the active application program via the operating environment. The 
advanced user interface may also include a set of utility programs such as a pop-up keyboard or an image 

magnifier utility. , 

FIG. 1 is a block diagram showing the interaction of computer system software and hardware in 
response to actions of user using an integrated operating environment such as Presentation Manager. 

FIG. 2 is an architectural block diagram of a computer system in accordance with the present 
invention, depicting the memory having read therein a series of application and operating system programs. 

FIG. 3 is a block diagram showing the interaction of computer system software and hardware in 
response to actions of a user designed in accordance with a first embodiment of the present invention. 

FIG. 4 is a flow diagram of the operation of the first embodiment of the present invention in response 

to a voice input signal. 

FIG. 5 is a diagram of an application profile. 
--FIG.-6-is -a-block-diagram-showing-the -interaction-of.xomputer_sy.stem_sbitware_and hardware in 



response to actions of a user designed in accordance with a second embodiment of the present invention. 
FIG. 7 is a flow diagram of the operation of the second embodiment of the present invention in 

response to a voice input signal. 

FIG. 8 is a flow diagram of the operation of a computer module which distinguishes between touch 
input meant to emulate a mouse pointing device and touch input meant to be a gesture or character. 

FIG. 1 shows the interaction of the user and the computer system utilizing application programs A, B, C 
and D 50-53 in a typical integrated operating environment such as Presentation Manager 56. 

The user inputs data into the computer system via device A (keyboard) 32. device B (mouse) 34 or 
touch sensor 36. As the user inputs data, the selected device generates an interrupt to activate the 
appropriate driver. In the case of the keyboard 32, driver A 60 is activated; in the case of the mouse 34, 
driver B 62 is energized; and input to the touch sensor 36 activates the touch driver 63. The appropriate 
driver signals the integrated operating environment 56. The integrated operating environment 56 keeps track 
of which application program is active and sends all input data to that application program. Alternatively, the 
user can change application programs, and the integrated operating environment 56 will route the message 
to the new active application program. In one preferred embodiment, application program A 50 is a 
spreadsheet program, application program B 51 is a word processor, application program C 52 is a drawing 
program, and application program D 53 is an electronic mail facility. Keeping track of the active program is 
a "focusing function", as the integrated operating environment 56 determines the "focus" of the user at a 
given time For purposes of illustration, assume the user is inputting data via the keyboard 32 and the 
spreadsheet 50 is the active application program. The spreadsheet 50 then decides what action should be 
performed. Generally, this action will fall into one of three categories: the spreadsheet 50 will change the 
display 28 the spreadsheet 50 will communicate with other I/O devices such as a printer, or the 
spreadsheet 50 will communicate with other applications. All of the messages are handled by the integrated 
operating environment 56. 

While the prior art integrated operating environments allow easier access to multiple application 
programs, and therefore, the user can select the particular application programs which best suit his needs, 
the user can only communicate with the application program with the input devices for which it has been 
written For example, spreadsheet 50 is a program which has been written for a combination of keyboard 32 
and mouse 34 Although a touch driver 63 can be developed to communicate coherently w.th the integrated 
operating environment 56, by converting input signals generated by the touch sensor 63 into mouse 
movements and mouse clicks, the capabilities of the touch input device 36 are limited by the repertoire of 
the mouse device. New categories of touch commands such as handwriting recognition or gestures cannot 
be easily accommodated by this method, especially by an application program such as spreadsheet 50 
which has no understanding of what the types of touch commands mean. Other new input devices are more 
drastically limited. Thus, although the system designer or user would like to include new user-friendly input 
devices into an individual workstation, he may find it impossible or severely limiting to use existing software. 

The preferred embodiment of the invention comprises a set of computer programs for controlling the 
interaction between a user and a computer system as shown in FIG. 2. The invention is primarily envisioned 
for use with a personal computer such as the IBM PS/2TM; however, the principles of this invention can be 
u i~~.~r ^rAroecinn cwQtAms Thft architectural 
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block d.agram of FIG. 2 includes a central processing unit (CPU) 20 connected by means of a system bus 
22 to a read-only memory (ROM) 24 and a random access memory (RAM) 26. Also included in the 
computer system in FIG. 2 are a display 28 by which the computer presents information to the user, and 
plurality of input devices including a keyboard 30, and devices A 32 and B 34, which for the purposes of 
illustration in the following description are a second alphanumeric keyboard 32 and a mouse 34 but which 
could be a specialized keyboard such as a numeric keyboard, and a remote pointing device or a variety of 
other input devices. The computer system further comprises a touch sensor 36 or touch input device for 
use at the surface of the display 28, a voice sensor 38 and an image sensor 40. The I/O 42 can be 
connected to communication lines or other I/O equipment. 

The random access memory (RAM) 26 includes application program A 50, application program B 51, 
application program C 42. and application program D 53. Examples of typical application programs would 
include word processors, spreadsheets, graphic programs, electronic mail, and data base programs The 
RAM 26 also stores an operating system 54, such as DOS™ or OS/2™, and an integrated operating 
environment program 56, such as WINDOWS™ or Presentation manager™, which allows several applica- 
tion programs to be running concurrently. Other software includes driver A 60 and driver B 62 which 
interpret the electrical signals generated by the second keyboard 32 and the mouse 34 respectively for the 
operating environment 56. The touch driver 63. the gesture recognition unit 64 and character recognition 
unit 65 handle input signals from the touch sensor. Input signals from the voice sensor 38 and the image 
sensor 40 are processed by the voice recognition unit 66 and image recognition unit 68 respectively 
Although depicted in FIG. 2 as entirely software, driver A 60. driver B 62. touch driver 63. gesture 
recognition unit 64. voice recognition unit 66 and image recognition unit 68 can also be a combination of 
hardware and software. Finally, also resident in RAM 26 are the set of software modules which make up the 
advanced user interface 100. 



25 



First Embodiment 



FIG. 3 shows a computer system designed in accordance with the present invention. Rather than being 
connected to the integrated operating environment 56 via a standard device driver at least one of the 

30 plurality input devices is connected to the advanced user interface 100, via a recognition unit. In FIG. 3. the 
voice sensor 38 and image sensor 40 are connected to the advanced user interface (AUI) 100 via their 
respective recognition units, voice recognition unit 65 and image recognition unit 68. A keyboard 32, and a 
mouse 34, remain connected to integrated operating environment 56 via their respective drivers driver A 60 
and driver B 62, since many application programs recognize keyboard and mouse input. Finally, in FIG. 3, 

35 the touch sensor 36 is connected to both the integrated operating environment 56 via the standard touch 
driver 63 and to the advanced user interface 100 via the gesture recognition unit 64 and the character 
recognition unit 65. The touch driver 63, gesture recognition unit 64 and character recognition unit 65 
translate signals from the touch sensor 36 to produce gesture or character "tokens'* respectively. The series 
of positions of a finger or stylus on the surface of the touch sensor 36 can be recognized as handwritten 

40 alphanumeric characters by the character recognition unit 65, or as a gesture such as a circle of right hand 
arrow by the gesture recognition unit 64. The voice recognition unit 66 digitizes the sound received by the 
voice sensor 38 and performs pattern matching against previously defined sound patterns to produce voice 
"•tokens". Image recognition unit 68 processes information from the image sensor 40 and outputs image 
"tokens". A "token" is an input signal which represents a series of electrical signals collected from an input 

45 device. For example, the series of points collected when a user draws a circle on the touch sensor can be 
defined as the "circle token". Coupling the touch sensor 36 to integrated operating environment 56 allows 
the user to take advantage of application programs which are specifically written for the touch sensor 36 or 
other existing application programs for which only primitive mouse input signals are needed. Yet as the 
touch sensor 36 is also connected to AUI through the gesture recognition unit 64 and the character 

so recognition unit 65, application programs which do not recognize touch input can still accept gesture or 
character "tokens" due to the translation capabilities of the advanced user interface 100. 

In other embodiments, the keyboard 32 and mouse 34 to be connected to both the AUI 100 and 
integrated operating environment 56 as is the touch sensor 36. In still other embodiments, all input devices 
are attached directly to the AUI 100 alone. By using a keyboard AUI link, a user could change the normal 
55 typewritten commands for an application program to those which he finds more convenient to input or 
memorize. As the AUI 100 is written as an application program, it is able to use the communication and 
focusing capabilities of the integrated operating environment. 

Another nrofcrroH omhnrtimfint nf the nrftsont invfintinn nsfis a sinnlfi nointinn riftvirp rlrivsr fnnt 
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pictured) to combine the function of driver B 62 and touch driver 63 by accepting input from both mouse 34 
and touch sensor 36 as well as any other pointing device. This embodiment is particularly adapted to 
operate with the touch input device described in U. S. Patent Number 4,868.332. to E. Greanias. et al.. 
entitled "Combined Finger Touch and Stylus Detection System for Use on the Viewing Surface of a Visual 
Display Device", filed June 26, 1986. or the remotely sensed stylus described in commonly assigned patent 
application. Serial No. 264,409, filed October 28. 1988. entitled "Remotely Sensed Personal Stylus", by E. 
Greanias, et al., which are hereby incorporated by reference. The pointing device driver arbitrates between 
mouse input from the mouse 34 and finger and stylus input from the touch input device 36 and sends 
appropriate input messages to the integrated operating environment 56. 

Referring back to FIG. 3, the AUI 100 is further divided into several computer code modules. The 
environment link module 101 contains algorithms for the execution of the procedures of the AUI 100 and 
provides communication capabilities to the integrated operating environment 56. The Alternative Input 
Subsystem (AIS) 103 provides an interface description which allows the designers of new user friendly input 
devices, e.g., voice sensor 38, to seamlessly interconnect with the Advanced User Interface 100. The AIS 
103 translates the input signals sent by the input devices and their respective recognition units to input 
messages useable by the remainder of the Advanced User Interface 100. A part of each input message, 
such as a prefix GS/ to denote a gesture, will be devoted to identify the input device from which the input 
- S ignal--is-received.-ln-this-embodiment,-4he-AIS_103-also.time_stamps_.eachJnputjn . 
correct application program receives the appropriate commands. 

The interface profile module 104 is comprised of sets of application profiles 105 and the user profiles 
107, which are files which list input messages produced by the AIS 103 from the input signals received by 
input devices 36, 38, 40. mapped to keyboard, mouse or other commands which are usable by existing 
application programs, e.g., mouse clicks, keystroke messages, MACROS, utility programs, etc. The 
Advanced User Interface utility (AUI utility) 109 is a set of utility programs such as a pop-up keyboard or an 
image magnifier utility which can be used with AUI 100. As shown, the AUI utility 109 can be written as a 
separate application program, or set of application programs, so that it can use the message passing 
capabilities of the integrated operating environment 56 to communicate with other application programs. 

A few examples of utilities which might be found in the advanced user interface utilities 109 include a 
"pop-up" keyboard, a "pop-up" numeric keypad, an image magnifying utility and an electronic mail routing 
slip The "pop-up" keyboard and "pop-up" keypad are functions which are preferably designed to operate 
with a touch input device at the display surface. By issuing the proper command, e.g., a circle gesture or 
touching a keyboard icon, the keyboard or keypad will "pop-up" on the screen. A user can quickly enter 
columns of alphanumeric information directly on the screen by touching the keys displayed on the screen 
thus eliminating the need to divert their attention to manual keyboard. The space needed for the keyboard 
or keypad is quite small, and will not completely obscure the work area of the spreadsheet, word processor, 
etc which is the active application program. The image magnifying utility will magnify a rectangle of fixed 
size around a point at which the appropriate gesture was made. The utility allows very accurate positioning 
of a cursor in the expanded image. After stylus liftoff, the normal size display is restored, and the selected 
cursor coordinates are sent to the active application program. 

The flow diagram in FIG. 4 depicts the operation of the computer system in FIG. 3 in response to a 
voice input signal. The active application program is the spreadsheet 50, and presented to the user on the 
display 28 is an array of selected data concurrently with a menu bar of possible functions available to the 
user Rather than striking a series of keys on the keyboard, the user elects to input data into the computer 
system using another input device, the voice sensor 38. In response to the user vocalizing the word 
"GRAPH" at 111 the voice sensor 38 generates electrical signals and sends them to the appropriate 
translation device, the voice recognition unit 66. In response to these electrical signals, the voice recognition 
unit is activated at 113 and sends an appropriate input signal, a voice input "token", to the Alternate Input 
Subsystem 103 At 1 15. the AIS 103 forwards an input message to the environment link 101 which contains 
information on the time the input signal was received and which input device sent the signal. In response to 
the input message, at 117 the environment link 101 queries the integrated operating environment 56 as to 
which application program is currently active. After determining the active application program, spreadsheet 
50 at 119 the environment link 101 refers to the application profile 105 of the spreadsheet 50 for the 
command which corresponds to the input message "GRAPH" which will be recognized by the spreadsheet 
50 ie the menu selection -graph--. The environment link 101 then at 121 checks the user profile 107 to 
determine whether there is a higher priority command which would override the command from the 
application profile 105. The environment link 101 determines which command has higher priority at 123. 
The environment link 101 at 125 then sends the higher priority command, in this example, the menu- 

— me *~ >Kr» ^fanratoH nnorstinn flnvirnnment 56 which 
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routes the command to the active application program spreadsheet 50. The spreadsheet 50 at 127 takes the 
appropriate action in response to that command, in this case, graphing the currently selected data and 
changing the information presented to the user by the display 28 accordingly. 



Interface Profiles 

To understand the power and flexibility of the advanced user interface 100, the application profiles 105 
and the user profiles 107 of the interface profile 104 must be explained in greater detail. Both types of 
profile associate lists of input messages with a list of corresponding commands which are recognized by 
the appropriate application program, the environment link module 101, or by the integrated operating 
environment 56 itself. The command can be the name of a program, an instruction recognized by the 
integrated operating environment or a MACRO. A MACRO is a file which contains an arbitrarily long string 
of "events" and/or utilities. An "event", for the purposes for this specification, is an input message which 
results from a user action such as a single keystroke, a mouse button press, a mouse button release, a 
mouse double click, single menu selection, a single gesture or other "token", or a single accelerator key 
selection. An example of a utility is the pop-up keyboard. A MACRO may also treat other previously defined 
MACROS as commands. Accelerator keys are often a feature of the integrated operating environment 56. 
They specify a keystroke sequence which emulates a menu selection. 

For example, an application may specify that the DELETE key is the same as selecting CUT from the 
EDIT menu and that SHIFT-INSERT is the same as selecting PASTE from the EDIT menu. These two 
examples are, in fact, widely used accelerators. 

Each application program may have its own application interface profile 105, which contains mappings 
of input messages and their corresponding commands for each input device connected to the alternate 
input subsystem 103. For example, one application profile might contain input messages and commands as 
shown in FIG. 5. The sample application profile shows input messages mapped against the corresponding 
command together with comments for display to the user describing the user action needed to create the 
input message and comments describing the action the computer system will take upon receiving that input 
message when the given application is active. 

The same input message from the AIS 103 will generally invoke a different MACRO on different 
application profiles 105. For example, a certain application profile 105 may specify that a MACRO named 
"Cut/Paste" should be executed in response to a Right-Arrow gesture, while a different application profile 
105 may map the Right-Arrow gesture to the keystroke sequence "Form 201 (ENTER)". When the gesture 
recognition unit 64 detects a right arrow gesture from the touch sensor 36, it calls environment link 101 
which determines the application program which owns the currently active window. The environment link 
101 reads the corresponding application's profile 105. merges that information with the appropriate user's 
profile 107. choosing the command with higher priority, then issues the command to the active application 
program. The integrated operating environment 56 or to other AUI modules as appropriate. 

The user profile 107 contains commands which are common to ail or several of the application 
programs 50-53. A separate user profile 107 can be written for each system user. The advanced user 
interface can determine the active user during log-on procedures. Entries in the user profile 107 can be 
used to override those in the application profiles 105. This situation might occur when a particular user 
prefers to input data in a different manner from that provided by the application profile 105 written for a 
group of system users. The user profile 105 can also be used to provide a global default action, i.e., if a 
particular input message was not found by PM-Link 101 in the appropriate application profile 105.' Where a 
user profile 107 is used for a global default action, the application profile 105 which has a MACRO for a 
given input message can be given a higher priority than the default action. 

While the preferred embodiments of the present invention make a distinction between the application 
profiles 105 and the user profiles 107, in other embodiments, it is possible to write separate application 
profiles 105 for each user. Where the interface profile module 104 was organized in this manner, if an 
application profile 105 did not exist for a particular user, the environment link 101 would refer to a default 
application interface profile 105. The default application profile 105 would be provided with the application 
program, the input device or by the programmer configuring the computer system. As currently envisioned, 
an application profile 105 for a given application program would generally be provided with the application 
itself. Alternatively, all application profiles 105 as well as a general user profile 107 could be provided by 
the system programmer. For an unsophisticated user who does not wish to explore the power of the 
advanced user interface 100, the default profiles will usually be adequate. However, many users will wish to 
tsiinr the adwanroH ..cer intprfarp mo tn thfiir narticular desires and soecification. Therefore, it will be 
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necessary to either write or modify an existing application or user profile. 

The Advanced User Interface Control Panel, one of the Advanced User Interface Ut.ht.es 109. allows the 
user to tailor many of the AUI functions to create a user unique profile in an extremely efficient and user 
friendly manner. Although the advanced user interface 100 contains fairly sophisticated computer code, 
through the use of the Control Panel the user need not understand its workings. Among the functions 
offered by the AUI Control Panel is the ability to modify or create profiles. This is ^omplished by invoking 
the interface profile create/modify module of the AUI Control Panel by selecting PROFILE from the 
Control Panel menu. The Control Panel first queries the user the name of the profile to be modified. At this 
point, a menu of the appropriate profile is displayed to the user, including a list of desc np tans . <rf user 
actions e.g., circle gesture, "cut" voice command, etc.. against a description of the action taken by he 
computer i.e., the MACRO, in response to the user action. If there is no application or user profile 
associated with the name entered, the utility creates a new profile and displays a blank menu. The user 
moves the cursor through the menu to add or modify the information in the profile, and then invokes the 
record command function or the record MACRO function. For example, returning to the first item in FIG. 5. 
a single line of the menu for the spreadsheet application profile may read: 



20 



USER ACTION 


COMMAND 


CIRCLE gesture on touch sensor 


Invoke the pop-up keyboard 



If the user wants to change the input message from the circle gesture to a right hand arrow gesture, he 
moves the cursor to the user action field and selects "USER ACTION" from the menu.-The user is 
prompted to perform the action by which he wishes to invoke the pop-up keyboard and then "S-prompto to 
25 add a written description of the action to be displayed on the menu. Similarly, if the user w,shes to change 
the MACRO invoked by the circle gesture, he moves the cursor to the MACRO field and enters the name of 

the MACRO to be executed. , . , 

The user creates a new profile in a similar fashion as an existing profile is modified. First moving the 
cursor to a blank user action field, selecting "USER ACTION" from the menu, then when finished moving 
30 the cursor to the corresponding macro field, entering a MACRO name or other command and so forth until 
the profile is complete. Thus, it becomes a relatively straightforward procedure for a relatively unsophis i - 
cated user to construct the advanced user interface profiles for the application programs he has selected to 
allow him to input data into a computer system in the most advantageous manner. 

A MACRO can be created independently from the creation of a user or application profile In 
35 Presentation Manager 56, for example, there are three cooperating utilities which allow the user to create a 
MACRO, modify a MACRO and execute a MACRO. 

The Macro Creation Utility (MCU) is a Presentation Manager program which can be set up as a tutorial, 
a prompter, or a totally invisible utility. It can be called from the standard program loader or via a 
designated gesture or other input signal. Its job is to record the user's sequence of commands tor future 
40 playback. The user merely starts the MCU. performs the desired sequence of opmtans. then stops he 
MCU. The MCU then prompts the user for required information such as the name of the MACRO. Once the 
macro is named, the user can control the use of that MACRO by referring to it in any of his application 

^^^SSS^ (MMU) is a Presentation Manager program which allows the user to single 
45 step through a previously created macro. MMU displays an English language description of each step in the 
macro before it executes. At any point in the playback process, the user can modify one or more 
commands in the macro. The modification may be performed by editing the contents of the English 
anguage window (for experienced macro writers) or by turning on the "record" function to overwnte 
previously stored commands. The MMU also allows the user to easily test, debug and modify MACROs. It 
50 also lets the user build MACROS by modifying existing MACROs which may already do most of what the 
user needs 

The Macro Execution Utility (MEU) is a Presentation Manager program which plays back the macro at 
full speed The MEU could be called by the environment link 101 whenever the user issues the appropriate 
g tore Fo! example, if a left arrow gesture is recognized, the environment link 101 looks in the current 
55 User's Profile to determine if that gesture is "globally" inhibited. If not. the environment hnk 101 then 
determines the name of the active application and reads the corresponding Appl.cat.on Profile to determine 
uihat «« macro is linked to the left arrow gesture, for this application. The environment link 101 then 
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electrical signals and thereby invokes the appropriate device specific code 144 is invoked at 221. The 
device specific code at 223 then sends a series of input signals to the AIS 203 corresponding to the word 
"GRAPH" The AIS 203 translates the input signals as a series of keyboard or mouse input messages to be 
added to the single queue of OS/2 154 at 225. After timestamping the stream of input messages. OS/2 
passes them to Presentation Manager 156 which keeps track of the focus of the user and passes the input 
messages to the active application program, the spreadsheet 150. 

The PM-Link 201 intercepts the message to the active application program at 227 and refers back to 
the AIS 203. to determine that the true input messages are a set of voice input messages at 229 and sends 
the true input messages, the voice input, to the spreadsheet 150 at 231. 

After the PM-Link 201 transmits the true voice input message, the application program will respond with 
its own message at 233 and one of two things may occur. If the application does not understand the type of 
input message sent, it returns a "R0" message to the PM-Link 201 at 233, indicating that it does not 
understand the input message. The PM-Link 201, in response to an R0 message, would normally go to the 
interface profiles 204 to determine the corresponding command in the appropriate application profiles 205 
at 241 and user profile 207 at 243. However, since the input messages indicate that they are a voice 
message at 237. the PM-Link 201 first sends the input messages to the voice recognition unit 215 to 
receive the correct voice input token for the word "GRAPH". After receiving this token at 239. the PM-L.nk 
-201-first-looks -in -the- applied - 
command to the input message which will be recognized by the spreadsheet 150. i.e.. the menu selection -- 
graph-- Then the PM-Link 201 refers to the user profile 207 of the current user at 243. The PM-L.nk 201 
then determines which profile has the higher priority command for the spreadsheet 150 at 243. and then 
sends the higher priority command to spreadsheet 150 at 247. The spreadsheet then takes the appropriate 
action at 249, in this case, graphing the currently selected data and changing the information presented to 

the user by the display 28. . jp-- . 

If on the other hand, the spreadsheet 150 can decipher, hand or is otherwise aware of voice input, an 
"R1" message is sent to the PM-Link 201 at 233. The PM-Link 201 would then take no further action for the 
current message. While the spreadsheet 150 may be aware of voice input, it may also elect to ignore the 
voice input in certain circumstances, or periods during the application program. Alternatively the aware 
application 150 may understand some voice input, but not others, and send an R0 message back to the 
PM-Link 201 at 233 whenever it does not understand a particular voice input message. For those voice 
input messages for which it received an "R0" message, the PM-Link 201 would consult the voice 
recognition unit 21 5 and the interface profiles 204 as detailed above. 

In the embodiment of FIG. 6, the character recognition 21 1 and the gesture recognition unit 213 are part 
of the Advanced User Interface 200. When PM-Link 201 receives an input message which might be a 
character or gesture, it sends the set of points to the character recognition unit 211. If the character 
recognition unit 211 recognizes the set of points as an alphanumeric character, that character is identified 
and sent back to PM-Link 201 for further processing in the active application program. If the set of points is 
not recognized as a character, the set of points is sent to the gesture recognition unit 213 to determine 
whether the set of points is recognized as a gesture. If it is recognized as a gesture. PM-L.nk 201 sends the 
qesture to the interface profiles 204 to find the set of instructions to send to the active appl.cat.on program 
from the appropriate application profile 205 or user profile 207. If it is not recognized as either a character 
or a gesture, the PM-Link 201 decides that the input message was probably intended as a mouse message 
and generates a "mouse buttondown, mouse buttonup" message at the last point in the set of po.nts. 

As an alternative to the default user profile described above, where various modes are used to 
determine what mouse equivalent commands are generated by user actions on the touch panel, the present 
invention can include a computer module within PM-Link 201 to differentiate between touch .nput which 
emulates a mouse buttondown event mouse move event or a mouse buttonup event, and that touch input 
which is to be considered a gesture or character. This is done by using a time delay to differentiate 
between the two types of touch input. The Advanced User Interface 200 allows the user to touch the screen 
and move to the desired position before the application is notified of the event by placing all he points 
received from the touch sensor in a stroke buffer which either stylus liftoff or the stylus has delayed at a 
particular point for the set time delay. Once the user reaches the desired position and stops moving for the 
set time delay, the application program will receive the event through PM-Link 201 as a mouse event For 
example if the user stops moving for 200 milliseconds, a mouse buttondown - mouse buttonup event at the 
desired position is communicated to the application program. However, if the user starts to move again 
before the end of the set period of time, the event is not generated until the user once again stops and 
pauses for the period of time delays. The time delay is typically defined in milliseconds and may be varied 
K « ii ~~ ...;«* ...hi^h the i.-spr is intsractina. The time delay may be specified 
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by the user in the user profile 207 or may be altered by the application program which is utilizing the touch 

in, JL°L« e «^, hand ; 5 8 T 6r d06S TOt St ° P at 3 Particu,ar point for the s P ecified time delay period and 
.nstead l,fts the stylus off the touch screen, the Advanced User Interface 200 selects the set of input points 
as a candidate for character or gesture recognition. In other words, the time de.ay provide a wndow n 
toe m wh,ch gestures can be made. During this time, the Advanced User Interface 200 will on y aNow 

beTandidlrr "f T ** ^ ^ ^ ' iftin9 0ff ' ,he points code ted wi no" 
wm I H 9 f Ur6 ' buta mouse evenl - lf the 'ifts off before the time de.ay period expires, an 
attempt : will be made to recognize the set of input points in the character recognition unit 211 or We 

SET 7, as a 9esture or character lf rt is not recogni2ed ' h0Wever ' the n01 ™' ^ eZS 

sequence will be generated: mouse buttondown, mouse buttonup at the last input point in the set If it is 

srr refers to the interface profi,es 204 to send the ~ ~ 

belo^iS^^ 96StUre ^ 3 °° ^ ^ ° f thS t0UCh 134 * bribed 

mJLillo^™ 9 determin f m he wisnes to inv ° k e the pop-up keyboard utility in the AUI utilities 
module 209. draws a crcle of the face of the touch sensor 134 and lifts the stylus off before the end c the 

ZTJ^' VT S6nS0r 134 96nerateS 3 Seri6S ° f int6rruptS t0 the de " ice code 142 a 2 i 

te AKM^xlZr ^T * T A ' S 203 correspondin 9 to the set of points in the circle at 263. 

oio ill 1 J2 t? ♦ P S ' 9nalS t0 m ° USe inpUt meSSages to be handled b V t^ single queue in 
OS/2 154 at 265. The mput messages are sent to the active application program, the spreadsheet 1 50 by 
Presentation Manager 156, but are intercepted first by the PM-Link 201 at 267. The PM-Link 201 then 

qU6 wH th6 p ! 203 d6termineS that the " mouse " input messa ^ ^e actually touch input at 269 

When PM-Link 201 discovers the first touch input point, usually a touch down in the sensor 134 it refers 
to the user profile 207 for the time delay period selected by the user at 271. At 273, the PM-L 201 
collects the touch input points in its stroke buffer until the stylus liftoff event is detected. As the PM-Link 
201 collects the touch input points, each time the user pauses on the touch sensor within the stroke 

SI! LTVl Stabm : 3t 275 ' ™- Link 201 StartS ,imin 9 the period of Ability to determine 
whether the time delay penod has expired at 277. If the user moves before the time delay period has 
expired .the PM-Lmk 201 continues collecting the touch input points in the stroke at 273. If. however the 
penod of stab.lity exceeds the time delay, a mouse buttondown, mouse buttonup command is generated at 
fhJ'J? ! 6XamP US6r d06S n0t p3USe before the stylus is lifted - and therefore, the stroke is sent to 
£nt ^° 9n ' tl0n Unit 21 1 ^ 9SStUre reC ° 9niti0n Unit 213 for processin 9 * 279. If the touch input 

L f" reC09n,zed as either a 9 esture « character, a mouse buttondown, mouse buttonup 
command would be generated at 281. H 

After being recognized by the gesture recognition unit 213 as a circle gesture, the PM-Link 201 passes 
he circle gesture to the spreadsheet 150. The spreadsheet returns an "R0" message at 285. indicating that 
the circle gesture was not understood. Since the circle gesture was not understood at 287, the PM-Link 201 
refers to the application profile 205 for the spreadsheet 150 at 289 and finds no corresponding command 
for the crcle gesture. The PM-Link 201 then refers to the user profile 207 at 291 which *e 
corresponding command "invoke the pop-up keyboard". As the command from the user profile 207 has the 
highest pnonty at 293. the PM-Link 201 sends this message to the AUI utilities module 209 at 295 and the 
pop-up keyboard is presented to the user on the display. 

While the present invention has been described with reference to specific embodiments, it will be 
understood by those skilled in the art that changes in form and detail may be made without departing from 
he sp.rrt and scope of the invention. For example, while gesture and character recognition has discussed in 
terms o a touch mput device, another pointing input device such as a mouse or a graphic tablet can be 
utilized to send gestures or characters to the Advanced User Interface. In addition/while specific mention 
has been made of particular application programs, integrated operating environments and input devices the 
flexibility of the Advanced User Interface can accommodate both further advances as well as the existinq 
implementations of these programs and devices. The embodiments presented are for purposes of example 
and .(lustration only and are not to be taken to limit the scope of the invention narrower than the scope of 
the appended claims. K 



Claims 

1. An advanced user interface, for use with an intearated ooeratino environment caoable of runnino a 
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p,ura«ity of app.ication programs simultaneously on a computer ^^^SX^i 

br^^ /espo Jto actions 

t^S^"« -Pied to said input device which transiates said input signals to 

"pSs which contain mappings of said input messages against corresponding commands useable 
by at least one of said plurality of application programs; ; and fj| and said 

an environment link module coupled to sad alterna ve ^pu ^subsy jtem ad p ^ 

integrated operating environment, said ^ ro ^ rt J^^^Z ,n ,aW interface profiles for the 
said input messages against said transmitted by said input device, 

application program wh.ch was act,ve at the t me M»d hnpu tsg du|e ^ ^ access 

and sending said corresponding commands to an appropriate comp 

memory; . resnonse to user actions different from those user 

^^^T^^^^ — « * - aPPliC3ti0n 

-^ -advanced-user 

, -sSs^^ fflS -ace profiles further 

CO m^a^^^^ — 0 r PU t 1 la, against corresponding instructions 

useable by more than one of said plurality of appli cat ™ W™ 8 ^ at |east one o{ said Corresponding 

, J£L SSS onet said corresponding instructs contained 

instructions contained in said user prof, « » d ^ tt ' to Ih. input m««gB. 
profile of the active applicat.cn program has no ^oo^g ^ ^ devjces 

!0 6. The advanced user interface as reeled in one of Cla ms to . comprising e.g. a 

are coupled to said alternative input subsystem ™Ju * said plura of np & ^ ^ & 

touch input device, a voice recogn.tion dev.ee, an image recognition 

keyboard. . fl nich further comprises a second plurality of 

7. The advanced user interface ^ recited m Cla-m 6^ device ^ wherejn 

application programs were originally written. . interface in combination with an integrated 

40 9. A set of interlace profiles or use ^^l^Tprog^ simultaneously on a computer 
operating environment capable of running a plu ahty profiles compnsing mappings of 

system only one of which is active at a g,en t me sa d set of ,nt p ^ ^ ^ 

input messages derived from the input s.g nals of at leas^ t J£^ um £> by at least one of said plurality 
advanced user interlace mapped against «^^^ f ^ 8 p „ U cation programs causes said computer 
45 of application programs so that at ^^^^^So^ from user actions for which the 

Ration ^^^^^Z 9 ° SSrcomprise at least one app.ication 
10. The set of interface prof.les as recited in Claim 9. wn,c messages derived 

a 9 a,ns« corresponding oommands concise a user pro«,e wnicn 
, 1. The sat ol interfaoa p-oWes as recited n cum 9 or Qf devices 

integrated operating environment )W_™J|»,.™ -lBtac innut siQnals from atjeastone input device to input 
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messages, interface profiles which comprise mappings of said input messages against corresponding 

commands by at least one of said plurality of application programs, comprising the steps of: 

receiving input signals transmitted from at least one input device in response to actions performed by a 

user; 

translating said input signals into an input message; 

determining which one of said application programs was active at the time said input signals were 
transmitted; 

finding a corresponding command to said input message in said interface profiles for said active application 
program; and, 

sending said corresponding command to an appropriate computer module in said random access memory; 
said computer system performing a given action in response to user actions different from those user 
actions for which said active application program was originally written without change to said active 
application program. 

13. The method of operating a computer system as recited in Claim 12 wherein: 
said receiving and translating steps are performed by said alternative input subsystem; 

said determining step is performed by said environment link module by querying said integrated operating 
environment; and 

said finding and sending steps are performed by said environment link module. 

14. The method of operating a computer system as recited in Claim 12 or 13, wherein said finding step 
comprises the steps of: 

finding a corresponding command to said input messages in an application profile for said active 
application; 

finding a corresponding command to said input messages in a user profile for said user operating said 
computer system; and 

determining which corresponding command has higher priority. 

15. The method of operating a computer system as recited in one of Claims 12 to 14, wherein said 
corresponding command is sent to said active application program and/or to a utilities module of said 
advanced user interface. 

16. A method of generating a command to a computer system in response to user actions on a surface 
of a touch input device comprising the steps of: 

collecting touch input points as a user moves a stylus across said touch input device surface; 

in response to said user pausing at a given point on said touch input device surface, starting the timing of a 

period of stability in which said user does not move said stylus from said given point; and, 

in response to said period of stability exceeding a predetermined period of time, generating a mouse 

command to said computer system at said given point. 

17. The method of generating a command to a computer system as recited in Claim 16 further 
comprising the step of: 

in the absence of a period of stability exceeding said predetermined period of time before said user lifts 
said stylus off said touch input device surface, recognizing said touch input points as a gesture input 
message. 

18. The method of generating a command to a computer system as recited in Claim 16 or 17 further 
comprising the step of: 

in the absence of a period of stability exceeding said predetermined period of time before said user lifts 
said stylus off said touch input device surface, recognizing said touch input points as a character input 
message. 

19. The method of generating a command to a computer system as recited in one of Claims 16 to 18 
further comprising the steps of: 

finding a corresponding command to said gesture input message in interface profiles containing mappings 
of input messages against corresponding commands; and, 

sending said corresponding command to an appropriate computer module in a random access memory of 
said computer system. 



EP 0 394 614 A2 



FIG.1 



APPLICATION 
A 
50 



APPLICATION 
B 
51 



APPLICATION 
C 
52 



APPLICATION 
D 
53 



INTEGRATED OPERATING ENVIRONMENT 
56 




OPERATING 
SYSTEM 
54 



BIOS 
70 




DRIVER A 
32 



TOUCH 
DRIVER 
63 



| DEVICE B 34 



TOUCH SENSOR 



THIS PAGE BLANK n 



EP 0 394 614 A2 



LU g 

< 2 

^ LU 

CO 



o 



O 



0C 
O 

CO co 

LU 
CO 



LU 
O 

> CD 
LU 

o 



LU 
O 

>. < 
UJ 

Q 



A 




LU 


en 




o 




O 

o 


CO 
21 


CD 
CO 


> 


SE 





CM 
CO 



Q 




CC 




< 




o 


O 


QQ 


CO 


>- 













oQ 

CC CO 



3 
Q- 



CM 



8 



81 











—J 


CO 


CL 


CM 


CO 




Q 







CC 

LU in 

}— CD 

S o 






o 





< 

CC 

LU O 

> 
CC 
Q 



CD 



CD 




CC 




LU 


CM 


7> 


CO 


CC 




o 





CO 

X <° 
O CC 

o > 

I — CC 
Q 



LU 




CC 


CO 


STU 


:co 


1 GE 


LU 

QC 



CO 

LU ^° 
O O 

o ^ 
> 

CC 



oo 
LU to 
CP O 

< o 

2 LU 
~~ CC 





X 



CO 
CM 




2 




o 




1 — 








1 


CD 


o_ 




CL. 




< 






THIS PAGE BLANK <msfio, 



EP 0 394 614 A2 



CO 

O 
LL. 




THIS PAGE BLAMK (uspio, 



EP 0 394 614 A2 



FIG. 4 
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FIG. 7 
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